Two-dimensional image processing based on third dimension data

ABSTRACT

Systems, methods and computer readable media for two-dimensional image processing based on third dimension data are described. Some implementations can include a method comprising obtaining first image data having a plurality of pixel values and obtaining second image data corresponding to each pixel value in the first image data. The method can also include receiving an indication of one or more control points in the first image data. The method can further include selectively filtering the first image data based on a distance from the control point and on one or more data values in the second image data corresponding to the corresponding control point. The method can also include outputting the selectively filtered first image data.

BACKGROUND

Digital images may include a two-dimensional array of image data valuessuch as brightness and color. Selective processing of images (e.g.,spatially varying filter parameters or filter strength) can beaccomplished with conventional image processing software. However, suchselective processing can become cumbersome and time-consuming wheneverhigh accuracy is required.

In addition to the two-dimensional array, a third dimension (orn-dimension) array of data values can be associated with an image. Auser may desire to perform one or more selective image processingfunctions on the two-dimensional image based on data values in the thirddimension array.

Also, increasingly, digital images are taken with mobile devices such aswireless phones and tablet computers, which often have touch screeninput devices. A user may desire to perform an image processing functionvia a user interface configured for mobile devices and/or touch screeninput devices.

SUMMARY

Some implementations relate generally to image processing, and, moreparticularly, to methods, systems and computer readable media fortwo-dimensional image processing based on third dimension data.

Some implementations can include a method comprising obtaining firstdata having a plurality of pixel values and obtaining second data havingone or more values corresponding to each pixel value in the first dataand receiving an indication of one or more control points in the firstdata. The method can also include selectively filtering the first databased on one or more data values in the second data corresponding to theone or more control points. The method can further include outputtingthe selectively filtered first data.

The first data can include two-dimensional data and the second dataincludes depth map data. Alternatively, the first data can includetwo-dimensional image data and the second data includes data from adifferent radiant spectrum than the first data.

The selective image filtering can be performed based on values receivedfrom on-image user interface controls superimposed on a displayedportion of the first data, wherein each on-image user interface controlcorresponds to one of the control points. The selectively filtered firstdata can include the second data. The selective image filtering can beperformed based on values received from user interface controlsdisplayed in a place different than superimposed on a displayed portionof the first data.

Some implementations can include a system having one or more computersconfigured to perform operations. The operations can include obtainingfirst image data having a plurality of pixel values and obtaining secondimage data corresponding to each pixel value in the first image data.The operations can also include receiving an indication of a controlpoint in the first image data. The operations can further includeselectively filtering the first image data based on a distance from thecontrol point and on one or more data values in the second image datacorresponding to the control point. The operations can also includeoutputting the selectively filtered first image data.

In some implementations, the first image data can includetwo-dimensional image data and the second image data includes depth mapdata. In some implementations, the first image data includestwo-dimensional image data and the second image data includes image datafrom a different spectrum than the first image data.

The selective image filtering can be performed based on values receivedfrom on-image user interface controls superimposed on a displayedportion of the first image data. The selectively filtered first imagedata can include the second image data. The selective image filteringcan be performed based on values received from user interface controlsdisplayed in a place different than superimposed on a displayed portionof the first image data.

Some implementations can include a nontransitory computer readablemedium having software instructions stored thereon that, when executedby a processor, cause the processor to perform operations. Theoperations can include obtaining first image data having a plurality ofpixel values and obtaining second image data corresponding to each pixelvalue in the first image data. The operations can also include receivingan indication of a control point in the first image data. The operationscan further include selectively filtering the first image data based ona distance from the control point and on one or more data values in thesecond image data corresponding to the control point. The operations canalso include outputting the selectively filtered first image data.

In some implementations, the first image data can includetwo-dimensional image data and the second image data includes depth mapdata. In some implementations, the first image data includestwo-dimensional image data and the second image data includes image datafrom a different spectrum than the first image data.

The selective image filtering can be performed based on values receivedfrom on-image user interface controls superimposed on a displayedportion of the first image data. The selectively filtered first imagedata can include the second image data. The selective image filteringcan be performed based on values received from user interface controlsdisplayed in a place different than superimposed on a displayed portionof the first image data.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of example first image data and second image data inaccordance with some implementations.

FIG. 2 is a flow chart of an example method for two-dimensional imageprocessing based on third dimension data in accordance with someimplementations.

FIG. 3 is a diagram of an example computer system in accordance withsome implementations.

DETAILED DESCRIPTION

FIG. 1 is a diagram of example first data and second data in accordancewith some implementations. The first data 102 can include atwo-dimensional image array of image data values (e.g., brightness andcolor values). The first data 102 can also include a pixel correspondingto a selected control point 106 (or image reference point). The firstdata can also include pixels 108 that are a distance from the pixelcorresponding to the control point 106. It will be appreciated that afirst set of distant pixels (108) is shown for illustration purposes,but that image data may include other pixels at other distances from thepixel corresponding to a control point. Also, it will be appreciatedthat there may be more than one control point.

The second data 104 can include a third (or more) dimension of datavalues in an array corresponding to the data values in the first data102. The second data 104 can include a pixel 106′ corresponding to thecontrol point (and to the pixel 106 in the first data 102). The secondimage data can also include a set of pixels (108′) a distance from thepixel 106′ corresponding to the control point (and to the pixels 108 inthe first data 102).

The second data 104 can include a depth map. A depth map can be an imageor an image channel that contains information relating to the distanceof the surfaces of objects in the image scene from a viewpoint. Depthmap is related to, and may be analogous to, depth buffer, Z-buffer,Z-buffering and Z-depth. The “Z” in some of the above-mentioned termsfollows a convention that the central axis of view of a camera isreferred to as the camera's Z axis, which may be the same as, ordifferent than, the absolute Z axis of the scene of an image.

Depth cues can be obtained from additional time-of-flight camerasmounted with the main imaging camera, camera arrays (e.g., stereo),moving a single camera while in burst mode or video mode, a focus stack,or even a flash/no flash image pair.

Also, while examples are discussed herein in terms of actual depthinformation from a real world scene, it will be appreciated that animplementation can be used with computer generated graphics. Additionalinformation (e.g., depth) may be readily available and accurate incomputer graphics as it may be coming from a computer generated 3Dscene, for example.

Alternatively, the second data 104 can include image data from aspectrum different than the spectrum of the first data 102. For example,the second data 104 can include values representing one or more of gammaradiation, x-ray radiation, ultraviolet radiation, visible radiation (ifthe first image data is not in the visible spectrum), infraredradiation, terahertz radiation, microwave radiation and radio waves. Thesecond image data could also include data obtained by a neighborhoodoperation on the image, such as texturedness or noisiness. Also, thesecond image data could include data that may be unrelated to pixels orradiance in any spectrum, e.g., population density or other demographicor statistical data. It will be appreciated that the first data 102 caninclude one or more of the types of data mentioned above regarding thesecond data 104. Also, it will be appreciated that while the term “imagedata” is used for illustration purposes, the image data could includedata that is not necessarily visible, e.g., a depth map, radiation dataor other n-dimensional data. In general, the phrase “image data” refersto an array of data having one or more dimensions.

FIG. 2 is a flow chart of an example method 200 for two-dimensionalimage processing based on third dimension data in accordance with someimplementations. Processing begins at 202, where first image data isobtained. For example, first image data may be obtained from an imagesensor (e.g., camera) in a user device. Alternatively, the first imagedata may be obtained by receiving transmitted data from another deviceor system via a wired or wireless connection. Processing continues to204.

At 204, second image data is obtained. The second image data may beobtained from a second sensor (e.g., depth measuring device, alternatespectrum sensor for sensing one or more of gamma radiation, x-rayradiation, ultraviolet radiation, visible radiation, infrared radiation,terahertz radiation, microwave radiation and radio waves, or the like)in the user device. Alternatively, the second image data may be obtainedby receiving transmitted data from another device or system via a wiredor wireless connection. Processing continues to 206.

At 206, the first image data is selectively filtered (or processed)based on one or more selected control points (or image reference points)and/or data values from the second image data. For example, anindication of one or more control points can be received. As usedherein, control point can refer to an on-screen user interface elementthat indicates a location of where an image processing function will beperformed. Also, control point can refer to a location in a data array(e.g., in first data and second data) that corresponds to a selectedpoint in an image. The control point can have an effect on the imageprocessing function in two respects. First, the image processingfunction can be performed with respect to distance from the controlpoint (e.g., the image processing function can vary according to thedistance from the control point). For example, the intensity oramplitude of an image processing function can decrease as a function ofdistance from the control point.

Second, the image processing function can be performed based on imagedata from a pixel value in the second image data corresponding to thelocation of the control point in the first image data. For example, ifthe second image data is a depth map and the control point correspondsto a depth map value of 5 feet, an image processing function may be toblur all pixels in the first image with a radius proportional to the theabsolute value of the difference of the pixel's depth map value and 5feet.

Also, a combination of two or more image processing functions can beapplied at each control point and each control point can have adifferent set of one or more image processing functions that are beingapplied at that control point.

In other words, 2D selective processing can be performed based on a 3Ddata model. For example, the availability of the 3D data model canprovide improved approaches for selective processing with filters suchas blur, saturation, relighting, and selective white balance, amongothers. Also, it will be appreciated that the additional data could havemore than a single dimension, which makes the data model n-dimensional,where n>=3. In some implementations, the selective processing could beperformed based only on the data values from the second image datawithout regarding to the distance from the control point.

In general, filter parameters or filter strength can be spatially variedbased on the control point location and/or data values from the secondimage data. Systems and methods for providing control pointfunctionality (or image reference point functionality) and associatedon-image user interface controls are described in U.S. Pat. Nos.6,728,421; 6,865,300; and 7,031,547, which are incorporated herein byreference. Processing continues to 208.

At 208, the second image data is optionally included with theselectively processed first image data. For example, the second imagedata could be provided with the processed first image data to enablefurther downstream processing. Processing continues to 210.

At 210, the selectively processed image is provided as output. Theselective processed image can be displayed, stored and/or transmitted toanother system or device via a wired or wireless network. It will beappreciated that 202-210 can be repeated in whole or in part in order toaccomplish a contemplated image processing task.

FIG. 3 is a diagram of an example computer device 300 that can be usedfor two-dimensional image processing based on third dimension data inaccordance with some implementations. The computer device 300 includes aprocessor 302, operating system 304, memory 306 and I/O interface 308.The memory 306 can include an image processing application 310 and oneor more images (or data arrays) 312.

In operation, the processor 302 may execute the image processingapplication 310 stored in the memory 306. The image processingapplication 310 can include software instructions that, when executed bythe processor, cause the processor to perform operations fortwo-dimensional image processing based on third dimension data inaccordance with the present disclosure (e.g., the image processingapplication 310 can cause the processor to perform one or more of steps202-210 described above and, in conjunction, can access the first andsecond image data 312). The image processing application 310 can alsooperate in conjunction with the operating system 304.

The computer (e.g., 300) can include, but is not limited to, a singleprocessor system, a multi-processor system (co-located or distributed),a cloud computing system, or a combination of the above.

The user device can include, but is not limited to, a desktop computer,a laptop computer, a portable computer, a tablet computing device, asmartphone, a feature phone, a personal digital assistant, a mediaplayer, an electronic book reader, an entertainment (or computing)system of a vehicle or the like. Other examples of devices includecomputing and/or display systems built into windows, walls, furniture,glasses, goggles, wrist watches, clothing or the like. In general, anycomputing device capable of implementing one or more of the methodsdescribed herein can be used.

The network connecting user devices to a conversation server can be awired or wireless network, and can include, but is not limited to, aWiFi network, a local area network, a wide area network, the Internet,or a combination of the above.

The data storage, memory and/or nontransitory computer readable mediumcan be a magnetic storage device (hard disk drive or the like), opticalstorage device (CD, DVD or the like), electronic storage device (RAM,ROM, flash, or the like). The software instructions can also becontained in, and provided as, an electronic signal, for example in theform of software as a service (SaaS) delivered from a server (e.g., adistributed system and/or a cloud computing system).

Moreover, some implementations of the disclosed method, system, andcomputer readable media can be implemented in software (e.g., as acomputer program product and/or nontransitory computer readable mediahaving stored instructions for performing one or more image processingtasks as described herein). The stored software instructions can beexecuted on a programmed general purpose computer, a special purposecomputer, a microprocessor, or the like.

It is, therefore, apparent that there is provided, in accordance withthe various example implementations disclosed herein, systems, methodsand computer readable media for two-dimensional image processing basedon third dimension data.

While the disclosed subject matter has been described in conjunctionwith a number of implementations, it is evident that many alternatives,modifications and variations would be or are apparent to those ofordinary skill in the applicable arts. Accordingly, Applicants intend toembrace all such alternatives, modifications, equivalents and variationsthat are within the spirit and scope of the disclosed subject matter.

1-18. (cancelled)
 19. A method comprising: obtaining first data having aplurality of values and obtaining second data having one or more valuescorresponding to one or more values in the first data, wherein thesecond data includes computer generated depth map data from a computergenerated three-dimensional scene; selectively filtering the first databased on one or more values in the second data corresponding to areference location value of the first data; and outputting theselectively filtered first data.
 20. The method of claim 19, wherein thereference location value of the first data is received from a userinterface control shown on a displayed portion of the first data. 21.The method of claim 19, wherein the second data includes data obtainedby a neighborhood operation on the first data.
 22. The method of claim19, wherein the second data includes data unrelated to pixelsrepresenting radiance in any spectrum.
 23. A system comprising: one ormore processors coupled to a computer readable medium having storedthereon software instructions that, when executed by the one or moreprocessors, cause the one or more processor to perform operationsincluding: obtaining first data having a plurality of values andobtaining second data having one or more values corresponding to one ormore values in the first data, wherein the second data includes computergenerated depth map data from a computer generated three-dimensionalscene; selectively filtering the first data based on one or more valuesin the second data corresponding to a reference location value of thefirst data; and outputting the selectively filtered first data.
 24. Thesystem of claim 23, wherein the reference location value of the firstdata is received from a user interface control shown on a displayedportion of the first data.
 25. The system of claim 23, wherein thesecond data includes data obtained by a neighborhood operation on thefirst data.
 26. The system of claim 23, wherein the second data includesdata unrelated to pixels representing radiance in any spectrum.
 27. Anontransitory computer readable medium having stored thereon softwareinstructions that, when executed by the one or more processors, causethe one or more processor to perform operations including: obtainingfirst data having a plurality of values and obtaining second data havingone or more values corresponding to one or more values in the firstdata, wherein the second data includes computer generated depth map datafrom a computer generated three-dimensional scene; selectively filteringthe first data based on one or more values in the second datacorresponding to a reference location value of the first data; andoutputting the selectively filtered first data.
 28. The nontransitorycomputer readable medium of claim 27, wherein the reference locationvalue of the first data is received from a user interface control shownon a displayed portion of the first data.
 29. The nontransitory computerreadable medium of claim 27, wherein the second data includes dataobtained by a neighborhood operation on the first data.
 30. Thenontransitory computer readable medium of claim 27, wherein the seconddata includes data unrelated to pixels representing radiance in anyspectrum.